{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "`Learn the Basics <intro.html>`_ ||\n",
    "`Quickstart <quickstart_tutorial.html>`_ ||\n",
    "**Tensors** ||\n",
    "`Datasets & DataLoaders <data_tutorial.html>`_ ||\n",
    "`Transforms <transforms_tutorial.html>`_ ||\n",
    "`Build Model <buildmodel_tutorial.html>`_ ||\n",
    "`Autograd <autogradqs_tutorial.html>`_ ||\n",
    "`Optimization <optimization_tutorial.html>`_ ||\n",
    "`Save & Load Model <saveloadrun_tutorial.html>`_\n",
    "\n",
    "Tensors\n",
    "==========================\n",
    "\n",
    "Tensors are a specialized data structure that are very similar to arrays and matrices.\n",
    "In PyTorch, we use tensors to encode the inputs and outputs of a model, as well as the model’s parameters.\n",
    "\n",
    "Tensors are similar to `NumPy’s <https://numpy.org/>`_ ndarrays, except that tensors can run on GPUs or other hardware accelerators. In fact, tensors and\n",
    "NumPy arrays can often share the same underlying memory, eliminating the need to copy data (see `bridge-to-np-label`). Tensors\n",
    "are also optimized for automatic differentiation (we'll see more about that later in the `Autograd <autogradqs_tutorial.html>`__\n",
    "section). If you’re familiar with ndarrays, you’ll be right at home with the Tensor API. If not, follow along!\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import torch\r\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initializing a Tensor\n",
    "\n",
    "\n",
    "Tensors can be initialized in various ways. Take a look at the following examples:\n",
    "\n",
    "**Directly from data(list)**\n",
    "\n",
    "Tensors can be created directly from data(list). The data type is automatically inferred.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "data = [[1, 2],[3, 4]]\r\n",
    "x_data = torch.tensor(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**From a NumPy array**\n",
    "\n",
    "Tensors can be created from NumPy arrays (and vice versa - see `bridge-to-np-label`).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "np_array = np.array(data)\r\n",
    "x_np = torch.from_numpy(np_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**From another tensor:**\n",
    "\n",
    "The new tensor retains the properties (shape, datatype) of the argument tensor, unless explicitly overridden.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ones Tensor: \n",
      " tensor([[1, 1],\n",
      "        [1, 1]]) \n",
      "\n",
      "Random Tensor: \n",
      " tensor([[0.6687, 0.5417],\n",
      "        [0.4490, 0.6810]]) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "x_ones = torch.ones_like(x_data) # retains the properties of x_data\r\n",
    "print(f\"Ones Tensor: \\n {x_ones} \\n\")\r\n",
    "\r\n",
    "x_rand = torch.rand_like(x_data, dtype=torch.float) # overrides the datatype of x_data\r\n",
    "print(f\"Random Tensor: \\n {x_rand} \\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**With random or constant values:**\n",
    "\n",
    "``shape`` is a tuple of tensor dimensions. In the functions below, it determines the dimensionality of the output tensor.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random Tensor: \n",
      " tensor([[0.3080, 0.7010, 0.9123],\n",
      "        [0.5257, 0.6251, 0.6253]]) \n",
      "\n",
      "Ones Tensor: \n",
      " tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]]) \n",
      "\n",
      "Zeros Tensor: \n",
      " tensor([[0., 0., 0.],\n",
      "        [0., 0., 0.]])\n"
     ]
    }
   ],
   "source": [
    "shape = (2,3,)\r\n",
    "rand_tensor = torch.rand(shape)\r\n",
    "ones_tensor = torch.ones(shape)\r\n",
    "zeros_tensor = torch.zeros(shape)\r\n",
    "\r\n",
    "print(f\"Random Tensor: \\n {rand_tensor} \\n\")\r\n",
    "print(f\"Ones Tensor: \\n {ones_tensor} \\n\")\r\n",
    "print(f\"Zeros Tensor: \\n {zeros_tensor}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attributes of a Tensor\n",
    "\n",
    "Tensor attributes describe their shape, datatype, and the device on which they are stored.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of tensor: torch.Size([3, 4])\n",
      "Datatype of tensor: torch.float32\n",
      "Device tensor is stored on: cpu\n"
     ]
    }
   ],
   "source": [
    "tensor = torch.rand(3,4)\r\n",
    "\r\n",
    "print(f\"Shape of tensor: {tensor.shape}\")\r\n",
    "print(f\"Datatype of tensor: {tensor.dtype}\")\r\n",
    "print(f\"Device tensor is stored on: {tensor.device}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Operations on Tensors\n",
    "\n",
    "Over 100 tensor operations, including arithmetic, linear algebra, matrix manipulation (transposing,\n",
    "indexing, slicing), sampling and more are\n",
    "comprehensively described `here <https://pytorch.org/docs/stable/torch.html>`.\n",
    "\n",
    "Each of these operations can be run on the GPU (at typically higher speeds than on a\n",
    "CPU). If you’re using Colab, allocate a GPU by going to Runtime > Change runtime type > GPU.\n",
    "\n",
    "By default, tensors are created on the CPU. We need to explicitly move tensors to the GPU using\n",
    "``.to`` method (after checking for GPU availability). Keep in mind that copying large tensors\n",
    "across devices can be expensive in terms of time and memory!\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'GPU is not available!'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We move our tensor to the GPU if available\r\n",
    "if torch.cuda.is_available():\r\n",
    "    tensor = tensor.to('cuda')\r\n",
    "f'GPU is{\"\" if torch.cuda.is_available() else \" not\"} available!'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try out some of the operations from the list.\n",
    "If you're familiar with the NumPy API, you'll find the Tensor API a breeze to use.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standard numpy-like indexing and slicing\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor tensor([[0.1747, 0.1718, 0.0399, 0.1555],\n",
      "        [0.3100, 0.5237, 0.9654, 0.5703],\n",
      "        [0.0486, 0.1281, 0.1545, 0.4679],\n",
      "        [0.2910, 0.3719, 0.7798, 0.0642]])\n",
      "First row:  tensor([0.1747, 0.1718, 0.0399, 0.1555])\n",
      "First column:  tensor([0.1747, 0.3100, 0.0486, 0.2910])\n",
      "Last column: tensor([0.1555, 0.5703, 0.4679, 0.0642])\n",
      "After modified: tensor([[0.1747, 0.0000, 0.0399, 0.1555],\n",
      "        [0.3100, 0.0000, 0.9654, 0.5703],\n",
      "        [0.0486, 0.0000, 0.1545, 0.4679],\n",
      "        [0.2910, 0.0000, 0.7798, 0.0642]])\n"
     ]
    }
   ],
   "source": [
    "tensor = torch.rand(4, 4)\r\n",
    "print('Tensor',tensor)\r\n",
    "print('First row: ', tensor[0])\r\n",
    "print('First column: ', tensor[:, 0])\r\n",
    "print('Last column:', tensor[..., -1])\r\n",
    "# change the second column to 0\r\n",
    "tensor[:,1] = 0\r\n",
    "print('After modified:',tensor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Joining tensors\n",
    "\n",
    "You can use ``torch.cat`` to concatenate a sequence of tensors along a given dimension.\n",
    "See also `torch.stack <https://pytorch.org/docs/stable/generated/torch.stack.html>`__,\n",
    "another tensor joining op that is subtly different from ``torch.cat``.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Join column:\n",
      " tensor([[1., 0., 0., 1., 0., 0., 1., 0., 0.],\n",
      "        [0., 1., 0., 0., 1., 0., 0., 1., 0.],\n",
      "        [0., 0., 1., 0., 0., 1., 0., 0., 1.]])\n",
      "Join row:\n",
      " tensor([[1., 0., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 0., 1.],\n",
      "        [1., 0., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 0., 1.],\n",
      "        [1., 0., 0.],\n",
      "        [0., 1., 0.],\n",
      "        [0., 0., 1.]])\n"
     ]
    }
   ],
   "source": [
    "tensor = torch.eye(3)\r\n",
    "# join column\r\n",
    "t1 = torch.cat([tensor, tensor, tensor], dim=1)\r\n",
    "print(\"Join column:\\n\",t1)\r\n",
    "#join row\r\n",
    "t2 = torch.cat([tensor,tensor,tensor],dim=0)\r\n",
    "print(\"Join row:\\n\",t2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Here dim means the dimension to join"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Arithmetic operations\n",
    "\n",
    "#### Matrix multiplication\n",
    "\n",
    "using operator `@` for `matrix multiplication` for two matrix who can multipy\n",
    "\n",
    "```python\n",
    "A @ B\n",
    "```\n",
    "\n",
    "#### Element-wise multiplication\n",
    "\n",
    "using operator `*` for `element-wise multiplication` for two matrix who have the `same shape` or can be broadcast\n",
    "\n",
    "```python\n",
    "A * B\n",
    "```\n",
    "\n",
    "> where A & B are matrix.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0007, 0.0567, 0.0236],\n",
       "        [0.3137, 0.5800, 0.5915],\n",
       "        [0.0991, 0.0028, 0.0649]])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor = torch.rand(3,3)\r\n",
    "\r\n",
    "# This computes the matrix multiplication between two tensors. Thus, y1, y2, y3 will have the same value\r\n",
    "y1 = tensor @ tensor.T\r\n",
    "y2 = tensor.matmul(tensor.T)\r\n",
    "\r\n",
    "y3 = torch.rand_like(tensor)\r\n",
    "torch.matmul(tensor, tensor.T, out=y3)\r\n",
    "\r\n",
    "\r\n",
    "# This computes the element-wise product. z1, z2, z3 will have the same value\r\n",
    "z1 = tensor * tensor\r\n",
    "z2 = tensor.mul(tensor)\r\n",
    "\r\n",
    "z3 = torch.rand_like(tensor)\r\n",
    "torch.mul(tensor, tensor, out=z3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Single-element tensors** \n",
    "\n",
    "If you have a one-element tensor, for example by aggregating all\n",
    "values of a tensor into one value, you can convert it to a Python\n",
    "numerical value using ``item()``:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.1316494941711426 <class 'float'>\n"
     ]
    }
   ],
   "source": [
    "agg = tensor.sum()\r\n",
    "agg_item = agg.item()\r\n",
    "print(agg_item, type(agg_item))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**In-place operations**\n",
    "Operations that store the result into the operand are called in-place. They are denoted by a ``_`` suffix.\n",
    "For example: ``x.copy_(y)``, ``x.t_()``, will change ``x``.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "print(tensor, \"\\n\")\r\n",
    "tensor.add_(5)\r\n",
    "print(tensor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\"><h4>Note</h4><p>In-place operations save some memory, but can be problematic when computing derivatives because of an immediate loss\n",
    "     of history. Hence, their use is discouraged.</p></div>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Bridge with NumPy\n",
    "\n",
    "Tensors on the CPU and NumPy arrays can share their underlying memory\n",
    "locations, and changing one will change\tthe other.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensor to NumPy array\n",
    "\n",
    "```python\n",
    "torch_tensor.numpy()\n",
    "```\n",
    "\n",
    "which will return a numpy format array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t: tensor([1., 1., 1., 1., 1.])\n",
      "n: [1. 1. 1. 1. 1.]\n"
     ]
    }
   ],
   "source": [
    "t = torch.ones(5)\r\n",
    "print(f\"t: {t}\")\r\n",
    "n = t.numpy()\r\n",
    "print(f\"n: {n}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Attention!**\n",
    "\n",
    "Since they share memory, a change in the tensor reflects in the NumPy array.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t: tensor([2., 2., 2., 2., 2.])\n",
      "n: [2. 2. 2. 2. 2.]\n"
     ]
    }
   ],
   "source": [
    "t.add_(1)\r\n",
    "print(f\"t: {t}\")\r\n",
    "print(f\"n: {n}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NumPy array to Tensor\n",
    "\n",
    "```python\n",
    "torch.from_numpy(numpy_arry)\n",
    "```\n",
    "\n",
    "which will return a torch tensor format array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "n = np.ones(5)\r\n",
    "t = torch.from_numpy(n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Changes in the NumPy array reflects in the tensor.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t: tensor([2., 2., 2., 2., 2.], dtype=torch.float64)\n",
      "n: [2. 2. 2. 2. 2.]\n"
     ]
    }
   ],
   "source": [
    "np.add(n, 1, out=n)\r\n",
    "print(f\"t: {t}\")\r\n",
    "print(f\"n: {n}\")"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "1baa965d5efe3ac65b79dfc60c0d706280b1da80fedb7760faf2759126c4f253"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
